1. Field of the Invention
The present invention relates to technology for controlling array rebuild, and in particular to a technology for maintaining availability of data during a potentially failing array rebuild.
2. Description of the Related Art
In certain arrangements of Redundant Arrays of Inexpensive Disks (RAID arrays), facilities are provided for rebuilding data from a failed disk using data from other disks in the array. This is usually achieved by distributing (striping) copies of data from each disk across the other disks in the array, so that it can be retrieved and assembled together on a spare disk if a disk fails. More recently, the concept of a preemptive rebuild has been developed. In this case, a disk that is determined to be at the point of failure (as detected by, for example, a count of transient or recoverable errors that reaches a predetermined threshold that indicates an impending total failure of the disk) is preemptively copied over to a spare before the impending failure can occur.
During RAID array rebuilds, failing hardware and firmware may encounter repetitive errors that prevent array rebuilds from completing and that may result in loss of access to data. Stated in more detail, the disk drive module firmware may indicate a predictive failure as described above, on the basis of, for example, an error thresholding mechanism. This indication in turn may be used to initiate a preemptive data rebuild from the failing component to another component, in order to improve system availability by reducing the time taken to prepare the spare disk drive module, and also to reduce the possibility of data loss—if the preemptive data rebuild is done successfully, there is no need to use the conventional technique of reading data back from all the other array components.
If, during the preemptive data migration, an error occurs that causes RAID operation failures, the recovery may be compromised. This is because, unless there is some higher level of control, the system-specified recovery action may be to continue to retry, which will merely repeat the action that failed. Such a repetition may ultimately result in loss of access to data or even in loss of the data itself.
An example of one known approach to the problem of maintaining data availability during RAID array rebuild is to delay rebuild recovery until the controlling software indicates that it is ready to allow such recovery. This approach, although it may have acceptable results in some circumstances, depends on the rebuild action itself being the cause of the error. In one alternative, it may be possible to have a mechanism wherein array components can be ignored before they have been initialized. However, this is also prone to error as the drive that has been predicted to be at the point of failure for one reason may fail for a different, unforeseen reason. A further alternative would be to bypass failing disk drive modules using out of band signaling. However, this is expensive in resource use and may not be technically possible (as would be the case, for example, if communications were routed over long fiber channel arbitrated loop (“FC-AL”) cables). Using multiple controllers is also unlikely to provide any benefit because, as they are likely to be running the same code, they may be subject to the same errors. Furthermore, it would be prohibitively expensive to write completely independent solutions for use on different controllers.
It would thus be desirable to have a technological means for controlling RAID array rebuild, and in particular to have a technological means for maintaining availability of data during a potentially failing array rebuild, without incurring excessive additional development cost.